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ABSTRACT 

This study provides preliminary^^ evidence as to the 
validity of measures derived from the "Tests of Sclent if i.c Thinking" 
(TST) • The TST .and the Graduate Record Examination's (GRE) were^ 
compared with regard to their relationships to intiere^ts, • ' 
self-appraisals, and accomplishments of s1:udents during their first 
year of graduate work in psychology* Criterion vatiables were 
obtained from a questionnaire mailed to students hear the end of the 
spring academic semester. Difficulties in data analysis ; were created 
bothbj^jthi^. i tftm-^sampling character of the experimental Itest data, 
^an:d""By^« relatively small rate of return of questionnaires. It is 



—possrb"le7to show, however, that the correl"ations-"jdo-rexhibit:-^^ — ' 

f appreciable nonchance variance with regard to consistency across 

tests and across related .questionnaire variables, / The. GRE tests were 
; found to be more effective" than the experimental instruments in 
^ 'predicting quality of the department attended,. but\ the experimental 
tests were more effective in two other domains: sel;f-^appraisals of 
knowledge 0;f psychology and skills in psychological\ activities; and 
prof ess ional accomplishments such as research, publications, and' 
teaching* (Author/ROF) 
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^ Abstract 
- . The "Tests of Scientific Tfeinking" (TST) are free-response job-sample 
tests intended to measure, aspects of scientific problem solving that require 
some degree of imaginativeness and ingenuity and that might therefore be used 
as dependent measures in research on "creativity." The present study /Wss 
designed to provide some preliminary evidence as to the validity of measures 
derived from-these instruments. The TST and the GRE, tests were compared with 
regard to their relationships to interests, self-appraisals , and accomplish- 
ments of students during their first year of graduate work in psychology. These 
criterion variables were obtained from a questionnaire^ mailed, to students near 
the end of the Spring, 1975, academic semester* Subjects were students who had 
taken the tests in October of 1973 for a study of the psychometric properties of 
the expejrimental tests. 

Difficulties in data analysis were created both by the item-sampling char- 
acter of the experimental test data, and by a relatively small rate of return 
-o f -ques tionnalxes- f rom . s t uden tis ,wJio„were actually attending graduate school in 
psychology. The estimated correlations for individual variables, therefore, 
reflect. ^ substantial, ai^ttoiirit of. error and can be regarded only as suggestive. 
It was possible to show, however, that the correlations do exhibit appreciable 
nbnchance variance with regard to consistency across tests and across related 
questionnaire variables, 

■ Jhe means. standar(LJLev:ia,tJ,^^ reliabilities of the TST scores were 

in general about the same as those obtained in the psychometric study, as were 
correlations with GRE scores. The GRE tests were found to be more effective 
than the experimental instruments- in predicting' quality of the department 
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A Study of the Predictive Validity of 
* th'e Tests of Scientific Thinking* 

V The "Tests of Scientific Thinking"' (TST) are f r^e-response job-sample 
tests which simulate tasks that might he encountered by a behavioral scientist, 

These instruments are intended, to measure aspects of scientific problem solv- 

/ ' ' * " ' ■ 

■ ing that require some 'degree >o£ imaginativeness and ingenuity and that might ■ 

■■_/■■■'■■■ ■ 

therefore be used as dependent measures in research on "creativity,/" In a ' 

•t ,■■.■> • . ' 

previous report (Frederiksen & Ward, 1975) the tests and associated scoring 

■' _ ' - ■ ■ 

methods were described, and evidence as to their psychom^jtric properties was 



presented* The current report contributes additional evidence which is needed 

■ - ' / . ' ... ■ . ■ 

before the tests are employed as criteria in studies of creativity. 

Three kinds of validity are generally distinguished: content validity , 

^ construct validity, and predic'tive validity. Content validity is demonstrated 

by showing that a t4st elicits behaviors that are rept:esentative of the domain 

of criterion behaviors. In the case of . tests of scientifiic thinking, we lack 

" ™; ; ^ V . .. i 

the detailed knowledge of the criterion domain to make such a demonstration, 

/ ' ' ' ' ■ 

except in an/impressionistic sense (which tnay be little different from "face" 



validity) ./'Construct validity is demonstrated by showing that; correlations of • 
the test/scores with a wide variety of other measures are consistent with ^ 

theoretical^ or at least intuitive, expectations. A construct validity study 

■ / , ■ . / " ' ' . - ' ■ . ' • ■ 

■" /■ ' ■■' " . . " - " . " • ■ 

with/these tests is in progress and will be reported separately. Predictive 



validity, finally , requires (a) that one or more acceptable criterion variables 
be obtainable and (b) that the test being validated be administered wellt before 



/the ciriterion behavior is exhibited,, The present Study is a predictive study 



*This researbh was supported by the Graduate Record Examinations Board. 



at. least in. the sense of the ^±me relaitionships in data collection. The 
criterion variables were chosen becaajsie they' appear to be relevant to our 
evaluation of the Tests of Scieatific Thinkirijg, but they can in no sense be 
considered validated m-easures jpf "imaginative problem solving. In this sense,- 
the study is more like construct validation, slitice the "criterion" variables - 
are, in varying degrees, intuitively relevant to the question of what^ the tests 
measure. 

Summary of Previous Stuidy 

/ The four tests of scientific thli^king (Frederiksen & Ward, 1975) are 
Foi/mulatlng Hypotheses (FH) , Evaluating Proposals (EP) , Solving Methodological 
ProBlCTig^SMP) , and Measuring Constructs (MC) . Candidates are instructed to 
propose one or more solutions to each problem posed by the test items; they 
are asked to write not only the answer they consider best but also others that 
they think deserve cousideratioti. A scoring method is used in which the scorer, 
rather than making a subjective evaluation,, assigns each answer to one of the 
categories in a classification of answers to each problem. These categories 
are given scale values based on rankings by a panel of judges. The method per- 



mits six scores to be generated for each item: three are concerned with the 



quality of solutions offered^ — quality of Best response (based on candidate's 
choice of his best answer) , Mean quality of responses, and Highest quality 
response (based on the scale values); and three depend on counts of the number 



of ideas given — Nund?er of responses, number of Unusual respons,es , and number 
of responses that are botH Unusual ""and High in quality. ' — 
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'The tests were administered as part of a regular administration of the 

GRE Advanced Psychology Test, using an item-sampling procedure. The analysis 

^was intended primarily to evaluate the tests from the standpoint of their 

psychometric properties* Scores from six-item tests were sufficiently reliahle 

to justify their use in research, except that the Unusual-High Quality score 

was unsatisfactory in the case of .SMP and MC. The tests were judged to be of ' 

suitable difficulty, although SMP was found to be a bit too difficult for the l 

GRE candidate population. The standard ^rror of measurement for a-tnore select 

subsample was somewhat smaller than was that for the entire group , indicating 

■- . ■ . ' ' ^. . • ■ ' . 

that the tests could be used with more advanced students. 

A factor analysis of the intercorrelations of all the scores generated by 
the TST revealed three factors: A number-of-responses factor including number 
scores from all four tests, and two quality factors, one based on quality scores 
from FH and^ MC and the, other on quality scores from EP and SMP. 

Correlations with GRE scores were low, even when corrected for unreliabil- 
J^ty^;--numbei:L,scor^s-~^ware_-^even~less-closely--related— to~GR 

quality scores. It was concluded that a substantial proportion of the true 

yax±ance_in_-the--experlmental^«test-scoreS"-is— not— accounted— for by present^ 

.<4if/ ■ ■ ■ ■ 

■ , ■ ■•■ 

tests, and that the new tests warrant further study , .particularly of their 
predictive and construct validity. '* 

^^^thod * ^ ^ 

— The availability of TST scores for a sizable cohort of candidates for 
admission to graduiate school suggested the desirability of a follow-up study 
as one approach to the investigation of validity. The study was baSed primar- 
ily on 1,600 candidates, each of whom took a three-item>stersion of " one, of the 
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t.ests in accordance with the item-sampling plan/ -Each candidate was asked to 
respond to a questionnaire whose items ^elicited information about type of 
school and program attended and about interests, plans, self appraisals, and 
accomplishments during the first year/ of graduate study. 

Thus> the purpose of this investigation was to discover relationships of 

scores on the 'experimental tests to ite^iis of information obtained from the 

\. . ■ . ■ 

questAonnaire that . are relevant to test validity, to compare these gelation- ^ 
ships with those involving the GRE tests, and to see to wh'at extent the new 
tests might make independent: contributions to the prediction of nccompl Lshmcnts 
as indicated by questionnaire responses. 

It was recognized that self^repor.Gs based on questionnaire items after y 
only one year- of graduate "work are inadequate as criteria of success^ and that 

n ■ : ■ ' ■ ■ ■ . • • • • 7 ■ 

simple dichotomizations^ cannot fully capture the valid variance on which 
answers to questionnaire iteuis are based. Furthermore, the assumptions involved 
in the item-sampling procedure ar^ not likely to be fully satisfied for the 
follow-up sample. The most important of these assumptions is that the subgroup 



I 



taking each of the many subtests required for .item sampling is a random sample, 
and thus representative of the total group. This assumption is more likely. to 
be violated as selection, including self--selection, operates to reduce the size: 
of the subgroups^ Despite these difficulties, the number and diversity of 
individuals availa^ make, this sample a unique riesource with which to 

search for relationship> . At the least, it was hoped, suggestive evidence . 



should be obtained which might later be confirmed through studies involving 
administration of full-length tests to more homogeneous groups of students./^ 

■ 8, 



iERJC 



1 



The Questionnaire , 

A copy of the "GRE Questionnaire" is included as Appendix A, The firist 
six items, intended for all recipients of the questionnaire, made it possible 

9 - ■ 

to compare grjDups that differed with regard to graduate school attendance,, 
''type of graduate program, and reasons for nouattendance. 

The remaining items were to be answered only by those who attended a 
graduate program in psychology' during the 1974-75 academic year. These items 
were concerned primarily with- intended areas of specialization, activity pre^f-^ 
eretices^ self-appraisals, and accomplishments during the year. Candidate^ 
were also invited to report the name of the university they were attending, . 
which made Xt possible to generate variables describing the quality of the 
pfbgram. These variables are based on published information concerning 
accreditation (APA, 1974) and departmental evaluations (Roose & Andersen, 1970). 
The student reports of their accomplishments (Item 17) are perhaps the most 
important from the standpoint of predictive validity. 



Forty-two variables were generated from the questionnaire responses. 
Table 1 provides the names of the variables, the range jof values pdssible for 

Irisert Table 1 about here \ V 

each, the questionnaire item (or items) on which each is based, and the method 
of translating responses into a numerical code. All are coded so that high ., 
numerical values represent the positive or *^hifeh** ei^d of the scale, \ 

" In this and subsequent tables, the questionnaire variables have been 
groupied into six general areas: indices of student and departBTent'^^arity ; 
program emphasis; areas of professional interest; preferred professional 
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activity; self-appraisels ; and professional activities and accomplishments 
during the academic year. 



Description of the Sample 



\ 



The ntunber of complete-data cases obtained in the previous investigation 

was 3,586. the sample reduction thab occurred at various stages is shown in 

r ' ' 

Table^ 2% ^ . v 

' ■ . " j . ^" • , - ¥: 

Insert Table !2 about he?e 



:ted into- three ^categories. Category 0 
through Item 6; students in this cate- " 



: ,The returned questionnaires were, so 
contained questionnaires completed only ^ 
gory either did not attend graduate school or were in fields other than psy-. 
chology proper (including, for example, guidance and counseling). Category 1 
subjects did' attend graduate school in psychology; these were the subjects used 
in the main analysis. Category 2 subjects attended graduate school, not in 
psychology but in f ields related to psychology, bu^ nonetheless comp let ed the 



entire questionnaire • 

As Table 2 shows, questionnaires were mailed to approximately' 90% of the 
original complete-data sample jj and presumably received by 75% of that group. 
Fifty percent of those who received questionnaires returned th^m completed ; 
However, of those who retu^ed questionnaires , only 49% met the criteria for 
•inclusion in the main group; 44% did not attend graduate school, or attended 
in an area unrelated to psychology, while 7% were in fields oiily related to 



psychology/ The number of Category 1 students was 654, Of these, 403 were 
from the intercorrelation sample in the previous study (th^e who took one item 



from each of two different tests) and 251 were in the reliability sample (those 
given three items from a single test). This latter sample is the most useful 
in the present study, since for these individuals ,it is possible to estimate 
relations based on three-item rather than one-item tests. Thus the various 
sources of reduction combined to leave available data tha^t are sufficiently 
abundant to justify analysis but prolpably not representative of the original 
group • 

Table 3 shows the means and staiidard deviations of GRE scores for the 



Insert Tabiy 3 about here 



three groups of students who returned questionnaires as well as for several 
subgroups that did not provide data. Among those who did respond , Category 1 
students — those who attended graduate sch^ool in psychology— were clearly the 
most capable in terms of GRE scores, with ^means only a little below 600, 
Cabagory 0 students, who either did not attend or attended a nonpsycholbglcal 
program, were substant:^ally lower, and thos^ in Category 2 (programs related 
to psychology) were still lovjer. The differences are substantial for all the • 
GRE scores,. but are smaller for GRE-V than for^GRE-Q and the Advanced Psychology 
Test, probably indicating a lesser emphasis on the. use of the Verbal test ,in 
graduate school selection. . \ 

Among students on whom follow-up data were not obtained, the means for the 
**Not Deliverable** group are only a little loyer tiian the- average for all those 
who returned questionnaires.. The remaining two groups— thbse. who presumably 
received questionnaires but chose not to return theinu . and those (*'Not Sent") 
who did not grant permission to be contacted in the rpllow-'Up — were 



substantially, lower , indicating, some self-selecti'on on al;)ility 'among those wlio ' 
cooperated in the- study^j - | 

' The last column of Table 3 contains mean scores for the original GRE . 



sample. The main 



follow-up sample had substantially higher means (as might be 



expected, sinqe (3RE scores are used in selection of giraduate students), but « 
was not notably^ ^restricted in the range of seo^esX^ . ■' ■ ' 7; /; S 

- Table k sh^ws the means and standard deviations of TST sx:ores based on 



Iri^ert Table 4 about here \ ^ j'^p 



the better— <^r^m(>r-e—ai4vari^ed—s-t-udeu^te-#- 



presented in Table 5» Table 1 should be consul-ted in soudying this table 



Insert Table 5 about here ■ • 
because it provides information on how toe variables were constructed from 



data from the original GRE study and from the follow-up. Unlike results . 
involving the GRE tests, where the follow-up sample has substantially higher^ -y.^::-^:}^^^. 



average scores, the experimental te^t means for the two groups are quite 

similar. Only the GRE scores were used in selection. But there is a tendetioy ^11 

■ • ■ ^ . •■ . ■ ' .. ■ - •'■/■■■ . - ■. • --'ii 

for SMP means to be higher for the follow-up group, which is consistent with 

' \ -^-^ - . , ' \- 

the hypothesis that the difficulty level of this test is more appropriate for 



A description of the sample in terms of the quest^ionnaire Variables is - -'r-.:''-7Mk 



the questioinaire items. . Some highlights: ..Students in the main sample studled^^^^^: w^^^ 

(Category 1) ten4^ to aim for a Ph.D.; to earn high. grades; to prefer applied 

research, clinical practice, or teaching; to . rv^i^t'/rer- clinical practice when :y:'';'^'-i:i^^^ 



li 
mm 



asked to choose one area; to rate themselves hi -/h oh knowle of ; psychology 



and statistics and low on 'teacKing skills ; to have a f ellow^^ or assiatant- 
ship; and/. t'o ha\/je had more experience in research activities than in writing 
and edifying during their first y * / 

.Table 5 also includes meanis and standard deviations on questionnkire 
;-iteais' for Category 2 students, trogether with t^ tests showing the significance 

" •■ ■ ■ . .-v^f ■■■ .■. ■ ' 

of difference? between the two groups. - 'Many differences aire liighly signifi- 

/•■'■•,■'.■'■■•■',". ' • ■ ■ ■ ■ . ' » ■ ■■ ' . . ■ ' 

cant. The studen.ts .who are' in a field related to psychology reported lower 
undergraduate. CPA's, on the average, but similar graduate school CPA's, T^^^ ^ 

' ■ ■ - ■ ■ ^ ■■' . . . • • ^ .. -V , . ~ ^ • ' .k ' ■■ . ■ J, . , , ■ 

Category 2 students, le"ss often- planned to earn /5a Ph.D. , were more like^ly to 
\ be in an applied area, named clinical practice and: counseling as .preferred ^ 
activities more of ten and research: liess of ten , rated themselves lower in all 
the skills areas except clinical, and claimed fewet accomplishmeTits during the 



first year of . graduate study'. ' . ' ' V 

Obviously a great deal of selection has taken .place in- going from the 
sample employed in the original investigation to that available for, the present 
study/ The reliability sample, which provides the best data for present p ur-r 
poses, constitutes only 19% of the original GRE reliability 'sample. One result 
*isi^tct>nsiderably higher GRE scoi-es and another, apparently, is students who are 
more serious about a career in psychology., 

' Reliabilities of Scores for Six-lte1a^^£xperimenta!l Tests 

. Kciiabili ties of the scores from each of the four experimental tests were 
calculated using' data from subjects rn"l:he f<illow-'up study wiio had beeit^-mejnbers 
of the original reliability sample. The' coefficients are presented in Table 6, 

V' . ■ ■ ■ ^ - . , ^ 

• . -v.. ' Insert Table 6 about here ' 



along with the comparable estimates reported for the same scores in the previous 
study. 'These are lower-bound estimates of the reliability of a six-item test. 



based on ''Lambda 2" (Guttman, 1945) ;\ their/Computation is described in the 

statistical appendix to the*^original report. , 

With three exceptions, the reliabilities are higher. for the follow-up 
study, in somfiT cases 'substantially so. The. mean difference in coefficients ' 

V c/. 

between;, the"^ two studios is/{ll, while the median difference is .08. This result ' 

might be interpreted to indicate that the experimental tests are mbr^ appropriate 

for th'e more highly selected students* contributing data to the follow-up study* 

Ho^afeyer,, it is also possible that this result arises from the greater • variability 

.: ;x . : . ■ . -..■<> ■ ^ . ..^ 

in the Jollow-up data of the terms used in '^qalculating the coefficients , espe- 

: - : ■ ■ ( ■■. , - . • 

daily since the computation of Lambda 2 involves Jthe use of squares of interitem 
covariance terms. Both the' decreased representativeness- anci the smaller N^'s ^ 
of the subsamples^ in;^^the fbllow-up data might lead to their having greater vari- 
ability and thus to inappropriately *"hi^h estimates when their squares are, entered 
into the formula. ' 

In Table 7 are given coefficient alpha reliability estimates for the two 



' Insert table 7 about her^ " .■ 

._«_„^^ :„/_ 

studies. The. computaticfe of alpha- involves the use of interitem covariances 



rather than their squares , and thus should be less heavily influenceci by insta- ^ 
bility in.^ the^omppneiit terras . ^ On the average, these estimates for the two 
studies are quite similar. Their means differ by *01^ and their medians by #03 5 
13 are higher and 10' are lower in the follo.w-up data. 



In view of the rather erratic variatipn between the coefficients obtained 
in the two studies^ one should prob.ably withhold judgment about the reliability 
of any single score; but it appears that the experimental tests in general are 
as reliable for the follow-up sample as for the original group.. 

• . ' Correlations of GRE Scores with TST Scores 

Correlations . between experimental test scores and the GRE' scores are given 
in Table 8 for the follow-up study data and, for comparison, for the data from 

■ ^, Insert Table 8. about here 

the originar GRE study. In general, the estimat-es are similar for the two 

-.\ ■ , '■ . ■ A f ■ . ' . ■■ • ' ' , . 

studies/ Across all experimental test scores and all GRE stores, coefficients 

are slightly lower 'for the follow-up study, with 44 of the 115 coefficients 
higher and 67 lower. The mean dif/erence in' coefficients is ,03 and the median 
difference is .02.' There is some tendency for correlations involving the. 
number scores to be higher and correlations involving quality scores to be 
lower for the follow-up sample. But SMP is a striking exception; for SMP the 
quality score correlations are higher. These are the scores that in the orig- 
inal study had the highest relation to GRE tests- and therefore might have been 

most sus'ceptible to restriction of range because of indirect selection. The , 

' ■ ■ . y ' ■■ ■ ■ ■ . ' 'I 

fact that these correlations rturned out to be higher rather than lower than * / 
those originalfy obtained perhaps suggt ts thiat SMP functions relatively better 

! ■ ■ • 

than the other tests when a more select sample of students is inyolved. 
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Correlations of GRE Scores with Questionnaire Variables 
Table 9 shows the correlations between GRE scores and the 42 variables 



derived from the questionnaire • Samplp' size and item sampling are not matters 

Insert Table 9 about here 
of concern here, since all individuals in Category 1 provide data for the 



computation of these coefficients. The N^'s approximate 650 for correlations 
with the Advanced Psychology Test and its two subscores. Experimental and 
S6cial"Personality , and 525 for correlations with the GRE Verbal and Qu'ahtita- 

-tive Aptitude scores. ' ; • 

Fifty-five percent of the .coefficients in the table are significant at 
:the l%=l level. Th4 Advanced Psychology test and its Experimental subscore are 
in general the two best predictors, but the five GRE scores tend to be quite 

^similar with regard .to. their correlations with questionnaire variables. - 

An examination of correlations by area shows that the GRE scores are by 
far most consistently related to the loose grouping of questions which we have / 
identified as indices of studen>t or department quality. These include ques- ; 
tions having to do with the student's degree aspirations and academic achieve- i 

' " ' . ' ^ . ■/ ' ■ ^ '.; ■ , ■ ■• 

ment, with ratings of department quality, with indirect indiceis such as whethe^ 
or not the student received fellowship support during the year, arid even with^" 
reports of satisfaction ^ith the program. Forty-two of the 45 coefficients In 
this area are significant at the .01- level. Of particular interest are the/ 

relations of GRE. scores to , graduate grade-point average, since this is the ' 

■ _ / " , ' ' ■ ' ■ / 

single roost frequently used criterion for the validation of selection instruments, 



16 



The correlations range from .iS to .26,^ with the highest being that involving 
the Advanced Psychology Test, \ ■ ' '\ : 



In several additional areas, the GRE scores show a substantial number ofi . 
significmt. correlatio though^'without such overwhelming consistency* Iri 
terms of students' desired professional activities, they are signifi'cantly i ^ 
related to. the number of activities\ checked, to interest in'-^both applied aiid 
basic research, and to a lack of .interest in guidance and counseling. 'In self- 
apprais.als of. skills arid' knowledge relevant to psychology, they predict particu 
larly self-reported ability to eva^tuate and interpret research. Also,- higher 
GRE scores are "^associated with attending a program whose objectives are aca- . 
demic or research-oriented rather than to prepare practitioners • ^ 

Finally, there are two areas in which only a scattering of significant 
coefficients appear. In the domain of the preferred professional activity, 
only a lack of interest in guidance and counseling shows consistent relations 
tO'the GRE scores. In that of professional activities and accomplishments dur- 
ing the academic year, none of the questionnaire variables is consistently 
related to the GRE scores. 

Along with the 'large number of statistically significant correlations, 
however, it is important to note the generally low level of . prediction obtained 
from the GRE scores. With the large N's available, a "significant" correlation 
may be no larger^ than .10/ , . ' ^ 

To facilitate comparison of the GRE and TST scores as p^dictors, we have 
underlined all those corjrfelations in the table which are .25 or higher. This 
figure was chosen since, with the N^'s typical for the experimental test correla- 
tions to be reported below, a coefficient of .25 is required for significance at 
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the 5% level. Only 17 correlations between GRE scores and questionnaire vari- 
ables equal or exceed this cut-off level. Those are concentrated, with two 
exceptions; in the^portion of the table~representiiTg--lndices of student and 
department quality. Further, the bulk of these (10 of 15) have to do with the 
two objective indices of department quality — whether or not the department has 
APA accreditation and th^. Roose-Andeifsen^ra.tijig--and' might indicate only . the 

* ■ . . I 

effects of a .sel^ction process in' which students scoring higher on the GRE 
tests are more , likely to be accepted by better' psychology departments-. All in- 
all^ then, if Jthe qiiestion is one of substantial rather than merely statisti- 
cally* significant prediction, the GRE scores show'very little in icelation to ^ 
student interests and activities during., the first year of graduate school in 



psychology. ; ... 

Relations of'TST Scores with Questionnaire Variables * ' 

■ i _ . , , ■ V i ■ • ' 

* - ' *" CI ' ■ * . ' ■ - * . . 

- It was originally intended that correlations between TST scores and 
variables derived from the questionnaire would be computed using the estimation 
procedures for item-sampled data described in the earlier report on these tests 
(Frederiksen & Ward, 1975). These procedures require the use of ite'm variance 
and covariance terms analogous to those employed in^ reliability estimation* 

-However, as was seen above, reliability ' coefficients for * the experimental test 
scores were somewhat 'unstable, presumably because of small subsample sizes 

\coupled with the likelihood- that^ the assumption of random assignment to item- 
^am|)ling subgroups does not Hold. If^is therefore probable .that the estimated 
corr^a±ions would also be unstable,. Moreover, . no test of significance would 
be availa^ble, since the conventional formula for the standard error is not. 
apprppriateXf or these coefficients. 
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Because of these problems, an alternative method for computing correla- 

• • • , •■ ■( 

tions was chosen in which it was not necessary to use the item variance and 
covariance terms . The' reliaMTip^ appropriate data. 

Each subject in that sample took a subtest composed, of three items from one 
test. Since there are 20 possible three-item combinations of six items, very 
few subjects took/exactly the same test ; but if it is assumed that the ityems 
;;are interchangeable (after adj^usting for differences in item means), scores 
can be obtained and correlations computed based on whatever three-item test 
was. taken. The resulting coefficients may be subject to less fluctuation 
th an those derived'' from the estimation procedure,, and in any case they can be 
tested for significance by conventional methods. The correlational results to 
be reported below are based on such three-item tests;.^ - v > . 

Each item score was^adjuste^l by subtracting the grand mean obtained .from 

• ^ . '. ■ . ^ 1^ ■ ' \ . . * ■ ■■■^ 

the complete reliability sample in the original- study. Since sample sizes and 
item-sampling methods in that study give reasonable assurance of equivalence of 
ability for subgroups, this, procedure should provide an appropriate adjustment 
. for differences in item difficulties. After this adjustment, six test scores i 
were obtained for each subject by summing over the three\items from the test; 
he h^d taken. These scores were then; correlated with the\42 questidnnaire 
variables. . v • - . . , ? • 

,._The results are presented in Tables 10 through 15. In each table are 



Insert Tables 10 through 15 about here \ 

.\ 



given correlations for one of the six scores (e.g., quality of "Best" response) 

from all four of the tests. Coefficients sign^.ficant at the .05 level aire 

.» , _ . 

underlined. . - ^ - -"^ . 



A count of the number of . coefficients reaching the .05 level of signifi- 
cance is revealing. Over the three tables of -correlations with quality scores,;', 
a total of 21 coefficients out of the 462 displayed attain this level of. 
significance • Since the expected number of correlations "significant", by 
chance is 23, it is impossible to conclude that any real relations are repre- 
sented in these, tables. For 'the remaining three tables—scores for Number , 
number of Unusual ; and number of Unusual-High quality responses — ;the situation 
is slightly better; here. 37 of 504 coefficients, or 7. 3%, reach significance at . 
the;5%, level. There is evidently some, nonchance variatidn i,n this part of - the 
data; but it is heavily embedded in noise, and. a coefficient-by-coefficient 
discussion of these i:esults could not be justified. 

Note, hWever, tha jxist pYesent^^^^ 

on three-item tests. In all our'yyork to date, we have, assumed that in usi-ng - 
'^ihese tasts we would employ six items in order to have mor^ adequate reliability 
While remaining within, a reasonable time requirement. With the' greater reli a--! 
bility of\ a longer test, a larger number of significant coefficients would be 
e3q)ected. yXbe procedure for estimating correlations for a six-item test , 
described in Frederiksen and Ward "(1975), was applied to the data, and the 
number of resulting coefficients of .25 or greater was examined. For the quality 
scores, 29, or 6.3%, were of this magnitude ; for the cotint scores, 76, oi;; 15'.1%, 
were this lar^.. These results again suggest a' lack of real relationsl^lps 
between these indices of graduate school performance and indices* of quality on 
the experimental tests, while those scores which depend on counts of numbers of. 
ideas-do-show-more--relatipns than-can- be attributed to chance. . ^ . 



Tests of the ■ significance of the combined results for the four tests were 

made for average correlations based on £ to transformations. (McNemar, 1962)/ 

• ■ ■ \ ■ ' . , ■ , / ' ■ ■ ' ■ ' 

The four correlations in any row of the preceding tables may be viewed as four 

• .. . / 

'• ' ■ " ■ ' " ' ■ -/ 

replications of a relationship; they are independent' replications in the sense 
that the four coefficients J. are based on nonove'rlapplrig samples of subjects. 
The jithree scores based^ on counts' ( Number ; Unusual » arid" Ujgusual'-High) are _con- 
sidered together because the factor analysis performed in the earlier study 
showed that they loaded on one factor, in contract to quality scores which 
loaded on two factors. ■ . . ^ 

Table 16 provides a summary oi^signif icant coefficients obtained by this 
procedure. Number , number of Unusual , and. number of Unusual-High quality 

Ins^ert Table 16 about here 

responses, show, respectively , ' 7 , 9 , and 10 relationships significant at at 
least the .05 level, whdrie 2 for each score woulid be expected by chance; go \ 
it is possible to examine those that appear in the table -with assurance that 
most of them are real. ' • ' ' " 

There is evident ct)nsistency across the three scores in the location of 

. ■ / " . ■ ' , • - • 

■ ■ . / , . - J . . 

significant results within sections of the table. Of the 26 average correla- 
tions significant at at least the 5% level, 12 arie deriyed from questions we 
have grouped und.er "Professional Activities,'- Students scoring high on the 
experimental tests tend to, have engaged in more of these activities, and 
specifically to have attended professional meetings and subscribed to journals 



published, engaged "in collaborative research, provided advice on experimental 

... . ' ^ ■ - ; 

design and statistics to other students, and worked with laboratory equipin^ 



' Eight significant relationships are found within questions grouped under 
"Student /Department Quality." Students scoring high tend to have plans for J 
thei-Fh-r-D^— r-ather than a master ' s degree, to attend a department with a high 
Roose-Andersen quality index, and to have obtained their support through a 

. • ■ ■ ■ a. ' . ■ ■ . \ ■ ' ' 

psychology-related activity during the first year. Four more coefficients are 
found within, th6 section of the table designated "Self-Appraisal/* Students 
scoring high on number of Unusual responses generally rate ^themselves high; 
along withVthostj high on Number of responses , they claim knowledge...of experi-"^^ 
mental design ;^ and those high on number of Unusual ^Hig h quality ideas see 
themselves as \ having a. lower level of clinical skills^ Finally , there are 
'significant relationships between" number ".of Unusual-High quality ideas ^and two 
indices indicating enrollment in a program which emphasizes research rather 
than praictice. , ' ' ^ ' 

A similar analysis was carried out on correlations of questionnaire 
variables with the three quality .scores. In view of the chance number of 
significant relationships that had been seen for individual correlation coef- 
ficients, it was not expected that positive results would have been. found. 
And, in fact, only four Relationships reaching the .05 level were found, where 
the chance expectation would have been six. 

These results indicate some ability for certain of the TST scores to 
predict possibly important aspects of first-year graduate performance, in 
spite of their low reliability. Addftionially , they suggest a discrimination 
between the experimental tests and the conventional predictors as to what 
aspects of tl^s^performancei are predicted* Recall that, in examining correlar 
tions between the Advanced Psychology test and the questionnaire variables. 



substantial relations were found mainly with variables representing individual 
or departmental quality indices, while there was little relation ''to variables 
representing professional activities. 

' Discussion and Conclusions 

In some resp.ects the present data were disappointing. We had hoped, 
perhaps naively, for a larger return; 3,200 individuals had indicated at the 
time of thie original GRE testing that they were willing to be contacted in a 
follow-up; We had expected that, among those who did respond, a larger pro- 
portion would have met the criteria for inclusion in the main analyses to be 
coriducted; of those who did cooperate with.; the study , only 49% were graduate . 
students in psychology during the pa'stjyear. The difficulties in analysis 
raised^ by the loss of cases, particularly in *view of the problems created by 
the item-sampling origin of the^ data, a^re abundantly clear in the presentation 
of 'results. Our best validity information is based on the average validity 
of four three-item tests', and ,the three items not the same for all subjects. 
Tlie reliability of such tests would be substantially lower than those reported 

for six-item tests. I 

■ . . . . . . ■ I ' ■ — 

Given these severe limitations in the data, the relationships which do 
•'• ■ . . * . " '■ . ■ 

emerge are perhaps more encouraiging than could have been eacpected. We find 

- . . ' ■ " •' , . . * ' ■ ■ „ • • ' ,.■ 

that the experimental measures have some ability to predict indices of student 

and departmental quality, the are'a in which the GRE tests are iDost effective as 

predictors* ; In addition, there is evidence that the experimental tests are 

related to two clusters of variables — self-appraisals of professional skills 

and reports of professional activities and accomplishments-^hlch have some face 

validity as potential additional criteria for predictive efforts, but which are 



not: strongly related to the GRE scores. Moreover j the best scores from among 
t'ae .six experimental scores, in termsj of apparent predictive power, are those 
for the number of Unusual responses and the number of tfnusual-High quality 
'responses, which are in general the least reliable of the experimental test 
scores. Since these are the scores from the TST which have the lowest rela- 
' tions ,to GRE scores, the results suggest that whatever predictive ^validity " 
the experimental tests have is incremental rather than arising from an* overlap 
with the psychology achievement tebt . 

It should also be noted that the questionnaire variables which served as- 
criteria against which to validate the experimental test scores are themselves, 
in many cases, highly tentative and exploratory. Some of these variables, such 
as graduate grade-point average and number of publications , aifp* clearly rele-- 
vaht indices of* the quality of^the student *s performance in graduate work. 
Others, however, are indirect in their import — they are more like indices of 
styles or preferences than of levei of perfprtnance, as in the areas 
o^f professional interest. An absence of significant relations with many of 
these latter variables cannot be labeled a failure of either the conventional, 
or experimental tests. In the various analyses in which we Jiave counted the 
number, of apparently significant coefficients/ arid' compared these with the 



number expected oyer all 42 questionnaire variables, therefore, we have to. 

some extent erred on the conservative side in the claims made for the -test scores 

Going one step further, the data-^obtained from this questionnaire cannot" 
be considered as exhausting the 'set of relevant criteria_^gainst— which7;thentest 

■ ' / ' ■ • ■ ■•. ' ■ , ' ,. ■ ■ ' 

scores mi^ght be validated* Other aspects of first year graduate performance 

■ ' ■ . ' ■ ■ • / ■ ' ■ ' 

might be considered, and some :^of /the aspects whicTi were assessed could be mea- 

; : ■ ' M\ ^ /••■"■ ^ • : ' . '-'''.^ . ■ 

sured more sensitively than by self report. Further., the first year of graduate 



work is itself likely to be quite inadequate as an index of future scientific 
performance. 

For a variety of reasons, then, we have much less evidence than is needed 
to judge what the long-run potential usefulness of these, instruments will be. 
The present study suggests some kinds o^ perfonnance for wliich the tests are 
likely to be especially relevant, as well as some differences among the tests 
and their various scores as to which' aire likely- to be most effectived These 
suggestions will have to be replidated in more;. tightly controlled Investiga- 
tions^ which avoid some of the difficulties Inherent in the design under which 
this exploratory work was performed.- We remain hopeful that the; tests will 
eventually prove .valuable as instruments for use. in basic experimental studies 
- as prototypes for possible new Iteih types for inclusion in the GRE tests, ajnd* 
potentially for use in the training of students and. in the evaluation of 
.instructional programs. , . " c 
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, Footnotes ; - 

■''T^is procedure sacrifice's the opportunity to use intercorrelation sample 
data to* provide a partly independent replication of results obtained with 
.reliability sample dat an However, a preliminary analysisf in which the signs 
of correlations between items and criterion variables were examiiied for con- 
sistency, showed a difference in results for the two samples — many more 
apparently significant relationships to the criteria, than ^ould be expected 

by chMce were found in the reliability sample data, but only a chance number 

■ . ' ' ■ ' ■.<.,'■ 
appealed In the intercorrelation sample data. The reason for this difference 

is not clear • ; One might speculate that, since the students in the reliability 

samjple worked on three items from a single test, while the other sample had to 

read new instructions aft^r completing one item, the first, group may have had 

n better opportunity to develop an appropriate set or strategy for attacking ; 

those problems, which in turn might have made their scores more valid,- 
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Table.l 



Questionnaire Variables 



Questionnaire Variable \ 



Studerit/Dept. Quality 

2. Plans PhD rather than MA 

.*3. Undergraduate GPA 

4. Graduate GPA 

5. Attends accredited dept. 

29. Rates dept. high in quality 

30. Satisfied .with department 

31. Department quality index^ . 
32.. Support through psychology 
33. Support through fellowship^ 



. Ques- 
Rahge -^ion 
Number 



Comments 



OtI 
2.17-4.5 
2.17-4.5 

0- 1 

1- 5, ' 
1-3- 
1-5 

vO-i . 
0-1 



i 



4. 
, 5 
6 
15 
13 
14 
15^ 
16 
16 



- X=#l to 8, receded Y=4.883-.333X 
/'X=#l to 8, 'receded Y=4.833-.333X 

s 

X=//l to 5, recoded Y=6-X 
X=//l to 3, recoded Y=4-X 

1=#1 to 4, 0=#5' 
■ 1=//1, 0=//2 to 4 ' 



27- 

^ 28: 



Program- Emphasis 
Plans academic program 

Program objective^ 'practice 
Deemphasizes teaching ^ 



0-1 



0-1 

.0-1 



11 
12 



l=//2,4,. 6, 7, 9, or lA; 0=#1,3, 
5,8'; oiLjX) 

l=//3, Q=#l 

1=#2, 0=//l or 3 ' , -/ 



- Interest Areas 

7. Number of afeas 

8. Administration 

9. Applied research 

10. Basic* research 

11. Clinical practice 

12. Guidance and counseling 

13. TeacHing 



0-8 
■~0-l 
0-1 
0-1 
0-1 
O7I 
0-1 



8 

8a 
8b 
8c 
8d 
8e 
8f 



Numbed of activities checked- 



Preferred Area 

14. Administration 

15*. Applied research 

16. Basic research • 

17. Clinicsal practice 

18. Suidance and counseling" 

19. Teaching ' 



\ 



0-1 
'0-1 
0-1" 
0-1 ' 
0-1 
0-1. 



9-1 
9-2 
9-3 
9-4 
9-5 
9-6 



20. 
21. 
22. 
23. 
24. 
25. 
26. 



Self-Appraisal . • 
Mean rating 

Knowledge of psychology 
Knowledge of sCatistics 
Experimental design 
Research -interpretation 
Clinical abi-Hty 
Te ac hing *ab i 1 i ty 



1-5- 
1-5 
1-5 
1-5 
1-5 
1-5 
1-5 



10 

lOabc 

lOde 

lOfh 

lOik 

lOg 

lOj 



Variables .21 tdf^f). 
X=#l t-G-5, recodfed Y=5-X 
X=#l to 5^ recoded Y=5-X 
recoded Y=5--X 
recoded Y=5-X 
recoded Y=5-X 



X=//l to 5 , 

X=//l- to 5, 

X=//l~to 5, 

X=#l to 5, 



recoded ^=5-X 



Prof essional., Activities ! 

34. Number of activities ; 0-15 17 

*35. Meetings, subscriptions 0-1 17 ab 

36. Publications' 0-1 17cde 



Number of activities checked 



ERIC 
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Table 1 (continued) 



• Questionnaire Variable. 


Range 


Ques- 
tion 
Number 


Comments 

• • ■ l> - ' . ^ . -^^ — 


■37. 


Planned, did indep. research 


0-1 


17fg 




38. 


. Did collaborative research 


0-1 


17h 




39. 


Taught undergraduates 


0-1 


17ij 




40. 


Advised on statistics 


0-1 


17k 




41. 


Helped prepare book 


0-1 ■ 


171 




42. 


Worked with "equipment 


0-1 


17nmo 




1. 


Other - 
Has MA rather than BA 


0-1 


3 





^APA. accreditation is in clinical, couniseling, and/or scho,ol psychology. 

• . p ' * 

]^ ' ' ... . ^ • : x 

JNs. are smaller because' about 15% of the subjects did not indicate the 
school they are attending 1 Roose-Andersen rating: 5 = highest, 1 « not , 
rated. - 

^s are approximately 40% smaller because the "other" category was^ , 
eliminated. ^ ' 
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' Table 2 

Numbers and Percentages for Complete-Data 
and Questionnaire Samples 



CO. 
.1 -w 
E CO 
0'« 



N 



0) 

0) rH 

rH CO 
^5 Oi CO 



O 







•a 




















•a 






a- 


0) 




> 




rH 




•H 




•H 




rH 




CO 




0) 




S 


le 


^« 


le 






M-( 


Oi 


o 


1 


O 


6 








CO 


5^ 




^5 


CO 







1 










rH 


0) 




'S 




O 


4J 










CO. 








a 


;3 
















CO 


CO 






0) 


Ph 




cu 




H. 




o 


rH 








1 






f. 


. o 




■i- 








00 


CO 


^5 


CO 


5^ 


o 


CO 



Complete Data Sainiple 

.No Permission'^ to Follow-up 
Could Not Be Matched to GRE 
' Data Tapes for Address 

Total Mailed 

Returned as Undeliv^rable 

Presumably Delivered • ^ 

Not Returned^ 

Returned * > ^ 
Ahswered Items 1-6 Ohly,.(Cati 0) 
Answered All Items; Graduate Student in 
Psychology-Related Field (Cat. 2) 

Answered All Items, Graduate Student ^in 
Psychology (Cat. 1) ' 
IntercorrelatioH' Sample - 
Reliability; Sample 



3586 
330 


100.0 
9.2 








21 


.6 








3235 
546 


90.2 
15.2 


100 J 0 
16.9 






2689" 
1366 


75.0 
38.1. 


- 83.1 
42'^. 2 


100.6 
50.8 


/' 


1323 " 
579 


36.9 
16.1 


40.9 
17.9 


49.2 
21.5 


100.0 
43.8 


90 


2.5 


2.8 

\ 

• V 


3.3 


&,.8 


654 
403 
251 


18.2 
11.2 
7.0 


\. 

\ 

20.2 
12.5 
7.8 . 


• 24.3" 
15.0 
9.3 


49.4 
30.5 
19.0 













100.0 
61.6 
38.4 



^Including approximately 15 questionnaires received after analyses had begun. 



• ■ ■ Table '3 

% " . - , • 

Mean' GRE Scores for Groups of Returned and Non-returned Questionnaires 







Returned 


I 

■ ■ > ! 

Not Returned 






GRE Scores ' 


. ". 
Cat .' 0 


Cat. 1 ' 


Cat. '2- 


Not Ue- 
, liverable 


No Re- 
sponse 


Not Sent 


- Original 
GRE Sample 


. GRE-V ' 


M 


562 . 


5'85 


538 


561 


544 




"558 




sp 


96 


95 


100 


98 


99 


105 

r* 


100 




N 


m 


527 


' 72 


465 


,1,154 


282 


2,998 , 


GRE-Q 


M . 


554 ' 


585 


504 " . 


557 


533 


540 


S4Q 




SD 


114 


108 


125 


114 


120 


.117 


118 




N 


494 


527; 

/ 


72 . 


465 . 


1,154 


282 


2,998 


GRE-Adv. 


M 


V 

546 


■ 591 


531 . 


558 * 


539 


539' 


JJfm ■ 




SD 


87 


'88 


' 98 


.'92 


92 


92 


93 




N 


578 


653 


90 


546 . 


1,357 


332. 


3,560 


Exp, 


M 


' 54.4 


. 58.5 


52.0 


55.4 


53.6 


53,5 


54.9 




SD 


8.9 


9.1 


9.9 


9.4 


9.4 


9.8' 


"9.5 - 




N 


578 


653 . 


90 


546 


.1,357 


- 332 • 


■ . 3,560 


Sbc.-Pers.M . 




58.2 


53.8 


55.2 


53.9 


54.5 


55.0 . 




SD 


/ 8.5 


8.7 


9.2" 


9.0 


. 9.0. - 


8.9 


' 9.0 




N . 

. V 


578 


6^3 


90 


546 


l',357 , 


332 ■ 


3,560 ' 



/ . - ■ ■ ■ ( 



Catl 0 = nonattendance or not in psychology. 
Cat, 1 - graduate student in psychblogy. ^ 
. Cat. 2 graduate, student in psychology-re"fated field. 



. • ''Table 4 
Means and Standard deviations of TST Scores 

I 









in 


■ E? 


SMP 




I- Score 




Follow-up 
. 0 amp 16 


Original 
□ampis 


Follow-up 
Sample 


Original 
Sample" 


Follow-up 
Sample 


Original 

Sample.; 


Follow-up 
Sample 


Original^ 
Sample'l 


Best 


M ' 


19.4 

*l Q 


19.7 






16.0 
2.0 ' 


14.9 ' 
2.2 . . 


14..4 
4.4 


14-6- '.^ 

3.6 -s; 


tfean 




17.8 

J* U 


' 17.9 


18.0 
1.7 . 


■ 17.8 
1.9 , 


14,6 
1.9 


14.0 
^.0 


13.0 
' 3.1 


■ ■ ■ ' '^i 

13.7 i 

3.0 : 


Highest 


M 

on , 


' 22. 2~ 

J.J 


22.1 


24'. 2 

" * 1.0 . 


23-9 

1 0 
1.0 


17.8 

7.1 


17.2 
1,8 


17.4 
3.6 


18.0 1 
3.2 ■ f 


{'i",;-..," '.. . ■ ^ ■ 

Sluraber 


mT 

SD ' 


' .6 . 


.6 


4.2 

-.g . 


" .9 ■ 


2.5 
.6 


■ 2.4 ■ 
.6 


• 3.0- 
.9' 


2;8 vi 

.8 


Unusual 

>■.-■ ■. !.'"■ 


M 

SD " 


.6 

•3 . 


.6 ' 


1.2^ 
■ .5 


■ 1.1 ■ 


.6 

. .. ' .3 . 


.5 


.9 

' .4 


.8'! 

... .4 •;; 


[Jnusual- 
BHigh 


M 

SD. 


.3 

' ,2 


.2 
.2 


.2 

■ .2 


.2 

.2' 


^ .2 
.1 


.1 

■ .1 


. • .2 
.2 


" • -'..2 
■ .2 1 




57 


359 ■ 


65 


339 


• 66 , 


■ 309 ■ 


■ 63 / 


340 



Jj^ased only on "Reliability Sample"~those individuals completing three 'items from one of the ; 
exjperiraental -.tests. , , 

^fe-No;''Best'V- scote w^^^ : ■ ' '■XM 



. . • Table 5 

Questionnaire Means and S.D.'s for Graduate Student Groups 







Category 1 


Category 


2 








M 


ou 


N 


M 










Sfudent /Dent - OiiaH W 
















2. 


Plan<3 PhD rather than MA 


. .89 ' 


.31 


598 


• . .75 


.44 . 


68 


2,55* : 


3. 


Undergraduate GPA 


3.92 


.47 


637 


3.80 


.46 


89 " 


2.30* 


4. 


Graduate GPA 


4.20 


.33 


V 624 


4.21 


.27 


87 


. 31 


5. 


Attends accredited dept. ^ 


.51 


.50 


552 


.57 


.50 


79 


i.po 


29. 


Rates dept*. high in quality 


3.06 


1.21 


630 


2.91 


1.18 


"85 


1.10 


3o; 


Satisfied with department 


2.14 


.66 


63'3 


2.17 


.69 


89 


• 39 , 


31. 


Department quality index 


1.99 


1 .39 


541 


2.34 


1.60 


79 


1.85 


32. 


Support through psychology 


.62 


' .48 


638 


.44 


.50 


90 


3.21** 


33. 


Support through fellowship 


.35 


.48 


398 


.28 


.45 


40 


.93 




Program Emphasis 
















6. 


Plans academic program 


.37 


.48 ° 


591 


.05 


.21 


85 


10.62*** 


27. 


Program objective; practice 


.50 


.50 


633 


.91 


.24 


85 


12.40*** ' 


28. 


Deemphasizes teaching ■ 


.56 


.50 


633 


.39 


.49 ' 


88 


3.04** 




xni.c47csi. f\rccio 
















7 
/ . 




2.41 


1.'04 


640 


2.17 


1.20 


90 


1.80 . 


o ... 


/ AHm"? n "f c f"T" a f^Ti • 
• . AUUiJ.ll Xo I. J. dL.XV.'ll 


.14 


.34; 


640 


.18 


.38 


90 


.95 


9. 


Applied research 


.50 


^50 


640 


.23 


.43 


90 


5.46*** 


10. 


Basic research 


.35- 


.48 


640 


.10 


.30- 


90 


6.78*** 


IIV 


Clinical practice 


- .55 


.50 ■ 


640 


.48 


.50 


90 


1.24 . 


12. 


TSuidanc^' and counseling 


.24 


.43 


640 


.6a 


.47 


90 


8.40*** 


13. 


Teaching . 


.50 


i50 


640 


.32 J 


.47 


90 


3.37*** 




Preferred Area 
















14. 


Administration. 


.03 


■ - .18 


640 


.02 


.15 


90 


.58 


15. 


Applied research 


.18 


.38 


640 


.03 


.18 


90 


6.20*** 


16. 


"Basic research ' • 


.15 


.36 


640 


.01 


.11 


90 


7.63*** 


17. 


Clinical practice 


.42 


.49 


640 


.38 


.49^ 


90 


.73 


.18. 


Guidance and counseling 


.08 


' .28, 


640 


.44 . 


.50 


90 


6.68*** 


19. 


Teachinig: ' 


.13 


.33 


640 


.11 


.32 


90 


:.:..55:^ ■ 




Self Add raisal 
















20. 


Mean rating 


2.46 


.50 


640 


2,. 32 


' .61 


90 


2.08* 


21. 


Knowledge of psychology 


2.58 


.59 


640 


2.50 


.60 


90 


1.19 


22. 


Knowledge of statistics 


2.40 


^.83 - 


640 


2.12 - 


.95 


.90 


2.66** 


23. 


Experimental design^ 


. 2.43 


.84 


639 


2.15 


1.00 


90 


2.53* 


24. 


Research interpretation 


2.55 


.74. 


638 


2.19 - 


.88 


90 


3.70*** 


25. 


Clinical ability - 


.2i31 


1.41 


630 


3.07 


1.00 


90 


6i36*** 


2.6. 


Teaching ability 


2.20 


1.24 


636 


2; 06 


1.25 


90 


1.00 




Professional Activities 
















34. 


Number of activities 


4.05 


2.*^64 


640 


2.42 


2.03 


90 


6.85*** 


•35. 


Meetings, subscriptions 


.41 


.37 


640 


.41 


.37^ 


90 


b.oo ' 


36. 


Publications 


.11 


.22 


640 


.05 


.16 


90 


3.16** " 



Table 5 (continued) 



Questionnaire Variable 



M 



Category 1 
SD 



N 



Category 2 
M ■ SD ■ N 



37. PlannedV did indep. research 

38. Did collaborative research 

39. Taught undergraduates 

40. Advised on statistics 
Al. Hel{)ed prepare book 
42. Worked with equipment 



.46 
.44 
.26- 
.24 
.06 
.24 



.40 
.50 
.38 
.43 

:23 
.30 



640 

640' 

640 

640: 

640 

640 



.33 
.14 
.14 
.17 
.03 
.06 



.39 

.35 

.29 

.37. 

.18 

.15 



90 
90 
90 
90 
90 
90 



2.95** 

7.1/*** 

3.52*** 

1.65 

1.43 

9.11*** 



Other ' 
Has MA. rather than BA 



.18 



.39 636 



.33 



.47 89 



2.88** 



£ < .05 (all two-tailed tests) 

. £ < • 01 

*** 

£ < .001 



Table 6 ' . 

"Lambda 2" Reliabilities of Six-Item TST Scores 

■ / 













Score 






Test 


Study - 


Best- 


Mean 


Highest 


Niaaber 


Unusual 


Unusual- 
High 


FH 


Original 


.76 

. .50 


.70 
.62 


.85. 
.65 


.68 
.67- 


.27 
.53 


.48 
.46 


' EP 


Fol low-up 
Original 


_lb 


.69 
.61 


'- •53 . 
.51 


.68 
.73 


.74 
.55 


.46 . " 
.44 


SMP 


Followrup 
Original 


.48 
.30 


.58 
.46 


•70 
.35 


.68 

.61 . 


.62 

. .42 


.37 

.05 . 


MC 


Follow-up 
Original 


.81 

.68 . 


.85 
.77 


.78 

.71- 


.86 

•77 , 


.32 
.36 


.63 

' . 17 ■ 



^N's for the follow-up study are 57, 65, 66, and 63 for FH, EP, 
SMP , and MC, respectively, except that N's were lower for Best " 
because subjects sometimes failed to mark their Best response. 

No "Best" score was obtained for EP. 



- . ■ . ■ -3^ , ' ■ • ■ . i . ■ ■ " 

\ ■ • . r _ . .... ' ■ ' 

Table 7 - ' ./ - , \ : ; 

Coefficient Alpha Reliabilities of Six-Item TST Scores^ 







'. Score / . 


. Test 


Study 


Best . 


Mean 


Highest- 


Number 


I 

Unusual * 

7/ 


Unusual- 
High' 


FH 


Follow-up 
Original 


•57 

, -48 


.54 
.58 


' .68 
- .60 


.57 • 

. r64. 


-.08? 

.7 


.31 

' .40-' 


j 

EP 


Follow-up 


__b 


. .53 


.27 


.62 


.52 


.36 




Original 




.58 


.48 - 


..72 


/49 ■ . 

, 1 ' . 


..40 


SMP 


Fdllow-up 
Original 


.30 
.22 


.45 
.41 


.61 , 
.28 


.63,, 
!59' 




-.06 
- -.11 


MC 


Follow-up 
Original 


.77 
.65 


.76 

.'74 


.71 

.68 


.81 

■..77 


; : .18 

' . ;32 • 


.47 

;10 



are given in Tabie 6. 
^No "Best" score was obtained for EP. / 
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Table 8 ' 
Correlations of GRE and TST Scores 



Test Score 



1'^ Best 
!;! Meau 

■n . *h 

Num. 
, Unus. 
U-H. 

" " Mean 

a 

■EP Num. 
Unus, 
U-H 

Best 

Meifiui 

Num. 
Unus. 
; U-H 

Best 
•> Mean 
'jjg High 
& Num. 
? Unus. 
-. • U-H 



GRE Scores 



Follow- O'rigi- 
. up nal 



.10 
.19 
.29 
.37 
.11' 
..16 

.35 
.42 

' i 36 . 
.28 
.20 

.34 

,.60 
.65 
.27 
..24 
-.04 

.03 

:..01 

.03 
,14 

V35 
.27 



.34 
.31 
.38 
.25 
.13 
.19 

.37 
.47 
.35 
.13 
.15 

,39 

.39, 

.49 

.15 

.01 

.20 

.28 
.29 
.35 
.22 
.28 
.24 



Q ■ 
V Follow- Origi: 
' up nal 



.3.7 
.28 
.49 
.57 
.05 
.17 

.11 
.27 
.25 
.Jl 
-.06 

.48 
.45 
.48 
.07 
-.02 
.31 

.26 
.49 
.45 
-.11 

■.12. 
,23 



.34 
.28 
.37 
.35 
.20 
; .19 

.31 
.40 
.27 
.09 
.13 

.32 
.34 
.39 
.14 
.03 
.21 

.30 
.34 
.33 

. .05 
.10 
.21' 



Adv. 
Follow- Origl- 
up ■ nal 



.08 
.10 
.26 
-.32 
.12 
.08 

.31 
.41 
.21 
.20 
.26 

.36 
.52 
.57 
.17 
-.17 
.01 

.23 
.01 
.07 
.19 
.36 
.17 



.28 
.24 
.31 

.22 
.12 
.17 

.33 

-^■.42 
.29 
.07 
.11 

.40 
:45 
.53 
.13 
-.09 
.18 

.24 
.25 
.31 
.19 
.23 
.32 



Exp. 
Follow- Origl- 
up nal 



Soc.-Fers. v 
Follow- Origi- 
up ;nal£ 



-.02 
-.02 
.17 
.35 
.10 
.02 

,26. 

.36 

.23 

.24 

.45 

.34 
.37 
.51. 
.19 
-.07 
.04 

.19 
.02 
«04 
.14 
.27 
.13 



.24 
.20 
.26 
.21 
> .13 
.14 

.29 
.37 
.27 
.09 
.15 

.33 
.36 
.50 
.19 
-.03 
.21 

,c.24 
.24 

'■.25' 
.10 
.18 

^ ;26 



.16 
.19 
.35 
.32 
.19 

:.ii 



.30 
.36 

.01 



.37 

,.60 
.53; 
.08 
-.29 
-.10 

^■27 
.01 
.14, 
.29 
.45 
.22- 



.25:'^^| 

.24:1 

.3r'i 

.17/ -I 
.09 I 

.16 



.46 

■';46:-| 
.02 '■: 

^l4 v; 

.10 ■ 



bi"Be8t" score was obtained for EP. 
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lable 9 



Correlations of Questionnaire Variables 
. vitli GRE Scores^ 



Questionnaire Variable 



2. 
•3. 

4. 

5. 
29. 
30. 
31. 
32. 
33. 



Stiident/Dept . Quality 
Plans- PhD rather than >iA 
Undergraduate GPA 
Graduate GPA " ■ 
Attends accredited dept. 
Rates dept. ^high in quality' 
Satisfied with department 
Department quality .index 
Support through psychology 
Support through fellovrship. 



Prpgram Emphasis 
•6. Plansr academic .program 

27. . Program ^objective: practice 

28. Deemphasizes teaching 



; Interest Areas 

7. Number of areas 

8. Administration 
9.. Appll^ed research 

10. Basic research 

11. Clinical practice 

12. - Guidance and counseling 

13. Teaching 



Preferred Area 

14. Administration 

15. Applied research 

16. Basic research 
•17. — G 1-in i cal— p r a c t i ce 

18. Guidance and counseling 

19. Teaching . ' 



Self-Appraisal 

20. Mean rating . 

21. Knowledge of psychology 

22. Knowledge of statistics 

23. Experimental design 
24.. Research interpretation. 

25. Clinical ability 

26. Teaching ability 



34," 
35. 
36; 



. Professional Activities 
Number^of activities 
Meetings j subscriptions 
Publications! 



V 



Adv. 



Exp. 



.19 
.22 
.18 
.38 
.27 
.17 
.37 
.22 
.27 



.12 

.14 
.15 
.27 
.23 
.14 

,17 
,15 



.18 

.20 

.26 

.34 ' 

.18 

.10 

.30 

.27- 



.17 

.18 

.22 

.32 

.1-8' 

.09 

.28 

.29 

m 



.1:9' 

.03 



.15 
,17 

;o2 



,11 

,23 
,04 



.20 
.29 
.02 



.12 
/06 
.19 
.14 
.01 
.15 
.09 



.01 
.00' 
.12 
.12 
.06 
.21 
06 



.13 
.01 

,i6 
,17 
,00 
,22 
.16 



.12 
.01 
.18 
.23 
.08 
.23 
.16 



-.08 
.06 
.10 
-.03 
-.13 
.03 



,02 
.02 
.10 
.03 
.13 
.05 



.07 
,02 
.14 
■0.3 
.17 
,05 



.05 
.06 
.19 
.11 
.17 
.06 



.04 
-.02 
.07 
.05 
.15 

•ao 

-.02 



.03 
.08 
,21 
,01 
,12 
,17 
.03 



.17 
.14 
.15 

,11 
.23 
.12 
, 08 



.18 

.17 

.16. 

.13 

.25 

.20 

.07 



.01 
-.12 
.04 



.02 
,17 
,00 



.13 
-.02 
.16 




.19 
.03 
.18 



Soc- 
Pers- 



.17 
.19 
.i23 
._29 
.14 
.10 
■ 25 
.19 
.19 



.01 
.12 

,04 



.11 
.02 
.10 
.06 
.13' 
.14 
.13 



-.09 
-.04 
.07 
.08 
-.13 
.02 



.13 
.07 
.07 
.08 
.15 
.02 
.07 




.04 
■.01 
.12 
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Table 9 



Questionnaire Variable .. ,^ 


V „, 


Q 


Adv. 


Ejcp. 


Soc- 
Pers. 


37. 


Planned, did Indep. research 


-.03 


-.05 


,01 


.04 


-.01 


38. 


Did collaborative research 


.06 


.04 


.13 


.17 


^ .05 


39. 


^.Taught undergraduates 


.07 


.03 


.11 


.13 


,.07 


40. 


Advised on statistics 


.01- 


.09 


.02 


.04 


-.03 


.41. 


Helped prepare book 


.12 


.05 


.05 


.05 


.03 


• 42. 


Worked with equipment 


.00 


.13 


.09 


.18 


-.04 


1. 


Other 

Has MA rather than'BA ' 


-.07 


-•19 ' 


•11 


.06 


.14 . 



^or correlations. with. V and Q scores, N,s are usually about 525^ and 
for the Advanced Psychology Test scores they are usually about 650* With 
, N of 525, a porrelition of .09 is significant at the 5% level, and' one of 
.11 is significant at the 1% level. Coefficients of .25 or higher are 
under lipied. V ^ ' _ 



Table 10 



Correlations of Best Quality Scores 
with Questionnaire Variables 



34. 
35. 
36. 



Professional Activities 
Number of activities 
Meetings suBscriptjipns 
Publications ' i 





(questionnaire variaoie 


rri 


*EP^ 




Mr 




' ■ \ ■ ■ 










2. 


Pian«3 PhD rather than MA 


M4 




-.08 


-.07 


3. ' 


llnH pr rjfidiia t* G HtPA- 


-.10 




.27 . 


-.04 


4, 


Graduate GPA - \ ' 


-.08 




.03 


.01 


5, 


Attends accredited dent. 


.01 




- .03 


.07 


29. 


T?fltp^ dpTit^ ■ hi i?h in ' anal i tv 


-.11 


mmmm 


.15 


-.05 


30, 


Satisfied with departments^ 


-.20 


mmmm 


.02 


-.06 


31. 


Dpnarttnpnt final 4 tv- index 


.03 




.32 


-.01 


32. 


SuDoort throuffhV DsvcholoEV 


.15 


MM 


.14 " 


.07 , 


33. 


Sunn or t thr ouGih f e llowshiD 


.08 - 





-.08 


-;06 




Program Emphasis 








-.04 ■ 


6. 


Plans academic program 


-•09 


~ 


.10 


< 27... 


Program objective: practice 


-.02 




-.09 


.10 


28. 


Deemphasizes teaching , 


-.08 




■■-.10 


-.07 




Interest Areas 






. . . ■ , 




7. 


Number of areas 


.01 


MM' - 


. .22 


-.00 


8 


Admi ni Q trat" "f nn 


-.15 


- 


.03 


.03 


9. 


Applied research 


..-.08 


M — 


.16 


-.07- 


10. 


Basic research 


.02' 





.09 


~-.oi 


11. 


Clinical practice 


.25 





.01. ■ 


' .04 


12. 


Guidance, and counseling 


-,13 





-.23 


.03 




Teachit\g^ _ ^ 


;-01 




.21 


.15 . 












-.04; 




AumxnisLraLiou 


.00 




-.18 


1 ^ 


Appiiea researcn 


.'13 




-.14 


-.16: 


16. 


Basic research 


ao 




.12 


.17 


17. 


Clinical practice 


.14 




.03 


.16 


18. 


Guidance and counseling 


-,11 




, -.03 


-^.19 


.19. 


Teaching 


-.08 . 




.14 , 


-.05 




V Self^Appraisal 










20. 


Mean rating 


,18 




-.10 


-.00 


21. 


Knowledge -of psychology 


-.11 


MM ' 


-.11 


-.09" 


22. 


knowledge -of statistics 


.26 




-.28 


.02 


23. 


Experimental design 


• 12 




-.02 


■ .07 


24. 


Research interpretation ^. 


. .09 




.14 


.10 


25; 


Clinical ability 


^ .35 




-.23° 


.04 


26. 


Teaching ability 


-.13 




.07- : 


-.17 



.02 
-.01 
.15. 



-.08 

-.06' 
-.10 



.09 
-.01 
-.03" 



m 



Si 



11^ 



-39- ' 
^able 10 (continued) 



Questibrinaire Variable . 


FH 




SMP 


. MC ' 




Planned > did indep* research 


• 03 




-.05 


-.00 




Did collaborative research. 


^.01 




-.09 


,10 


39. 


Taught Undergraduates ^ 






' .12 


.21 


'40. 


Advised on statistics 


.17 




-.07 


.02 


41. 


Holpdd prepare book - 


-.08 




.03 


..04 




W^^rked with equiptnen|| 


.01 




-.08 


.03 




Other- r 












Has MA rather than "BA 


-.00 




-.04 


-•.18 




a " 
Correlations based on 3-itcra 


tests. 


Coefficients ai8»iif icant at 


Che 


5% level are underlined* 












b 

No : "Best" score was obtained for'EP. 


s 
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Table 11 



Correlations of Mean Quality Scores 
with Questionnaire Variables^ 



Questionnaire Variable 



FH 


EP . 


SMP . 


f 


,08 


.31 


-.20 


-.00' 


-.04 


-.10 


.27 


.07 


.03 


. .32 


.03 


-.02 


.06 


.12 


.09 


.09 


^.11 


.01 


.14 


-.00 


-.22 


.08 


.17 


.03 


.13. 


.04 


.17 


.11 


.02 


.04 


.07 


.06 


.10 


.10 


-.19 


.10 



SCudent/Dept.' Quality 

2. Plans PhD rather Chan MA 

3. Undergraduate, GPA 

4. Graduate GPA 

5. Attends accredited dept. ' 

29. Rates dept. high in quality 

30. Satisfied with department 

31. Department quality index 

32. Suppdrt through psychology- 

33. Support through fellowship 



Program Emphasis 
6. Plans academic program 

27. Program objective: practice 

28. Deemphasiges teaching ,' 



.04 
.12 
.15 



.06 
.04 
.22 



Interest Areas 

7. Number of areas 

8. Adratnist-cation 

9. Applied research 

10. Basic research 

11. Clinical practice 

12.. Guidance ariid counseling 

13. Teaching 



,08 
.10 
.16 
.03 
.02 
.12 
.08 



,05 
,03 
,25 
,17 
.10 
.14 
.05 



Preferred Area 

14. Administration 

15: Applied research 

16. Basic research 

17. Clinical practice 

18. Guidance and counseling 

19. Teaching 



Self-Appraisal 

20. Mean rating ■ 

21. Knowledge .psychology 

22. Knowledge of statistics 
,23.. Experimental design 

24. Research interpretation 

25. Clinical ability 

26. Teaching ability 



Professional Activities 

34. Number of activities 

35. Meetings, subscriptions 

36. Publications 



,03 
.01 

.04' 



.01 
.09 
.07 
.07 
.01 
.18 
.05 



.03 

; 14 



-.02 
.05 

-.02 

-.07 
.05. 
.05 

-.04 



-.02 


-.28 


-.02 


.01 


-.13 


-.15 


-.11 


-.20 


.16- 


.10 


-.15 


.11 


.02 


..07 


.11 


.16 


.09 


.19 


-.04 


-.08 


-.14 


-.01 


.19 


-.06 


.10 


-.01 


.02 


-.17 


-.07 


-.13 


-.00 


: -.10 


.31 


.06 


-.03 


-.06 


-.09 


-.06 


-.01 


-.12 . 


.07 


.05 


.12 . 


-.07 


.18 


.10 


-.17 . 


-.09 


-.12 


-.03 


\. 11 


-.20 


-.06 


-.05 


-.06 


-^".01 


-.09 


.10 


- -.05 . 


-.17 


.08 


-.15 


- .00 


.01 



42 
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Table 11 (continued) 



Questionnaire Variable 


FH, 




' SMP 


: MC 


37. 


Planned, did Indep, research. 


-.02 . 


-.05 


-.06 


-.00 


38. 


Did. collaborative research 


-.10 


-.14 


-.14 


' .07 


39. 


Taught undergraduates- 


-.15 


-.00 


" 121 


.cm ■ 


40. 


Advised on statistics 


.12 


-.09 


-.04 


-.00 


41. 


Helped prepare book^ ... 


.06 


.15 .. 


■r.02 


<:oi 


42. 


Worked with equipment 


.00, 


• -.02 


-.15 


' .0?;: 


1. 


Other 

Has MA rather than BA 


■ 'i" ■ _ ■ 
.08 


.00 


.01 


-.17 



Correlations based on 3-ltein tests. Coefficients significant at 
the 5% level are underlined. 



m 



ERIC 



43 



-42- 



Table 12 



Correlations of Highest 
with Quei^tiohnaire 



Quality Scores 
Variables^' 



Questionnaire Variable 



FH 



BP 



SMP 



MC 



^ Student/Dept. Quality 

2. Plans PhD rather tlian MA 

3. * Undergraduate CPA 
4/ Graduate GPA 
5, Attends accredited dfept. 

29. Rates dept. high in quality 

30. Satisfied with department 

31. Department quality irid^ 

32. Support through psychology 

33. Support through fellowship 



/ 

/2 
<12 
.02 
.01- 
-.11 
-.31 
.11 
.18 
.16 




.08 
.03 
.07 
. 12. 
-.04 
'^.00 
.10 
.11 
.24 





Program Emphasis 


/' 








6. 


Plans academic program _^ 


-.08 


.06 


.09 


-.11 


27. 


Program objective: practice 


-.66— 


-.07 


-.07 


.05 


28. 


^P^janphasizes teaching 


'"/•ID 


• XD 


— • UO 


1 A 




Interest Areas 


/ - 








•7 

7. 


Number of areas. 




Cxi 
— • Ul 




in 

. J-U 


Q 

p. 


Administration 


• 11 


• ^1 


no 


( , . Uj 




Applied research 


— 9n 

• it u 




. UO 


OQ 

. U-7 


10. 


Basic research 


.02 


.12 


.07 


-.15 


11. 


Clinical practice / 


.13 


.06 


-.02 


.18 


12. 


Guidance and counseling 


-.02 


-.09 


-.22' 


-.00 


' 13. 


Teaching " 


.05 


-.02 


.08 


.05 




Preferred Area ' 










14. 


Administration 


.00 


-.03 


,02 


-.04 


15. 


Applied research 


-.17 


-.,23 


-.06 


-.14 


16. 


Basic research 


. 05 


.24 


: -.07 


-.09 


17; 


Clinical practice 


.10 ■ 


-.00 


-.03 


.25 


18. 


Guidance anid counseling 


-.03 


-.06 


.00 


. -.17 


19. 


Teaching/ 


. .04 


.10 


•17 


.07 




Self-Appraisal 










. 20. 


Mean rating 


.02 


.22 


.11 


-.14 


' 21. 


« Knowledge of psychology 


-.-22 


.16 


.12 


, -.09 


22. 


Knowledge of statistics 


.27 


.12 


-.02 


-.05 


23. 


Experimental design 


-•10 . 


.19 


.17 


-.07 


24. 


/'Research interpretation 


-.05 


• .12. 


.19 


-.18 


25/ 


>Clinical ability 


.22 


.02 


-.16 


.03 


■ 26. 


Teiaching ability 


-.03 


.18 


.02 


-.14 




, Professional Activities 










34. 


Number of activities 


-.04 


■.06 


;08 


• .05 


35. 


Meetings, subscriptions 


-.06 ~ 


-.'30 


.09. 


-.04 


36. 


Publications 


.00 


.07 


.11 


.'02 
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Table 12 (continued) 



. . . ■ / .■ . . — ; 

J Questionnaire Variab^le 


FH 




, : SMP- ■ 


•MC 


37. 


Planneii , did indep> research. - 


-.07 


.17 


-.07 


-.01' 


38. 


Did collaborative research 


-.15 


■ .22 


-.04.; 


.16 ' 


39. 


Taught undergraduates 


.00 , 


.04 V 


.25 


-.03 


AO, 


Advised on statistics 


.20 


. ■ :o2' 


.07 


.07 


41. 


Helped •prfepare book a 


.05 


-.03 


.04 


'.07 


42. 


Worked V7ith equipment 


-.03 . 


.09 


-.12 


■ .04 


i. 


Other 

Has MA rather than BA ' 


.11 


• .20 ~ 


.19 


-.12 



iia 
mm 



Correlations based on 3-itein tests. Coefficients significant'' at 
.the 5% level are underlined. 




m 

m 
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Table . 13 / 

Correlations of Number of Responses Scoreis 
with Quest ionniaire Variabi^s^ . 



. ^Questionnaire Variable 




EP 


SMP ■ 


MC 




' Student /Dep^^^ 




.22 






•2. 


Plans PhD rather than MA 


.19 




■ .15 


3; 


Undergraduate CPA 


-.01 


.16 


. • -.21 


.03 


4. 


Graduate CPA • 


^ .05 


. -.07 


■' -.02 : 


.15 


5. 


Attends acQredited dept . 


-.09 


" .30 ■:■ 


.17 • 


■ .08' 


29. 


Ra t e s dept. h i gh in q u a 1 i ty 


-.04 • 


-.05 - ■ 


>' .00 • 


.08 


30.. 


Satisfied with department ' 


-.18 


-.07 ' 


.02 


-.00 


31. 


Department quality index 


. ;.10 • 


• .24 


'25 


.18 


32. 


Support through psychology 


.16 


.16 


.26 


.08 


33. 


Support through fellowship 


.05 


.19 




.17 




Program EiTiphasis 










.6. 


Plans academic program _ 


-.08- 


.12 


.17 


-.18 


-~277- 


-Program obiec tive : practice 


.12 


-.16 


-.19 


-.06 


28. 


Deemphiasizes teaching. ' ' : 


- .0.4 


.27 


-.30 


.07 




Interest Ajreas 










7. 


Number of areas 


••'.io „ 


.04 


.03 


.15 


8. 


Administration 


.09 


-.05 


.06 


-.01 


9. 


Applied res'ea^^^ s . 


.05 


.00 


-.01 


.19 


10. 


Basic research 


-.11 


.15 . 


.06 


' ■ -.16 


11. 


Clinical practice 


.17 


-.07 


-.11. 


.25 




Guidance and counseling 


-.16 


.11 


.03 


r-.05 


13. 


Teaching ~ 


■ ' -26 


.00 


.04 


.04 ■ 














14. 


Adminiatr'atlon 


. 04 


.03 


.09 


-.05 


15. 


Applied research 


-.05 


-.21 


.05 


.19 


16. 


Basic research 


-.12 


. .26 


.05 


-.33 


17. 


Clinical practice 


.03 


-.10 


. -.07 


.15 


18. 


Guidance and counseling 


-.06 


.02 


.10 


-.17 


19- 


Teaching . 


.22 


.13 


-.15 


.11 




Self-Appraisal 






■ - , ' ■■■ ' ■ 




20. 


Mean rating 


-.05 


.12 


.18 


.03 


. 21. 


Knowletige of psychology 


-.21 


.23 


.22 


. .04 


22. 


Knowledge of statistics 


,.-.01 


-.03 


.03 


-.03. 


23. . 


, Expe rlmental design . 


.11 


.20 


.24 


.02 


24. 


Research interpretation 


-.08 


.03 


.13 


• -.16 


25. 


Clinical ability 


.01 


-.07 


.02 


.17 


26. 


Teaching abili ty , 


.02 


/.04 


-.03 


.10 




- Professional Activities 










34^ 


Number of activities 


. 06 . 


■ .15 


.32 


. .15 • 


35. 


Meetings i subscriptions- 


■ .00 


-.05 ,. 


.08 


.23 


36. 


Publications . 


-.05 


-.03 


.23 


.10 
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Table 13 (continued) 



Questionnaire Variable 


FM' " 


EP' 






37. 


: Planned, did indep. research 


-.02 


.24 


.21 


-.06 


38; 


Did collaborative research 


.01 


.:.i29,; 


.18 


.24 : 


39- 


Taught undergraduates 


.15 • 


-.02 


.23 


-1 11 


40- 


AdVised oh statistics 


.11. 


> .04 


.21 


.16 


41. 


Helped prepare book 


.03 


-,.17 


.12 


.10 


42. 


Worked with equipment 


. 01 


.19 


.03 


.05 




Other 










1. 


Has MA, rather thain BA - 


-.13 i 


V -.03 


.22 


.09 




a 

Correlatioas based on 3-item 


tests. 


Coefficients 


significant at 


the 


5% level are underlined. 











0 



ii 
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'■ ■ ■ . Table 14 .■ • 

Correlations of Number o^ Unusual Responses 

Si • 

with,, Questionnaire Variables 



■ - . .^^ ■ — — I — — • ■ 

Questionnaire Variable 


FH 


EP 


SMP : 


r - MC 


- ■ 


Student /Dept. Quality 








; • 


2. 


Plans PhD rather than MA 


;22 


^ .26 


.21 


.03 


*» 

J. 


Undergraduate GPA 


.08 


.09 


--25 ■ 


-. 06 


4. 


Graduate GPA 


.. .01 


-.24 


-.05 * 


, . 19 • 


5. 


Attends acctedited dept. 


-.09 


.27 


-.l95'< 


.04 


29. 


A.Ratfes dept; high in quality 


-.09 


.12 


^.12 


.06 


30." 


Satisfied with department 


-. 14 


-.06 


-.14 


.02 


31 . 


Department quaXity index 


-.29 


.25 


•^.03 


• 18 


32. 


Support through psychology 


.12 ■ 


>30 


.14 


• 04 


33. 


Stipport through fellowship 


- -,08 


.12 


-.22 


.01 




Program Emphasis 










6. 


-Plans academic program 


.02 


.23 


.06 


-•03 


27. 


Prograra objective: . practice 




1 1 


— . u/ 


— . UZ 


28. 


:Deemphasizes teaching 


• U/ 




— • z J 


91 ' 




Interest Areas 








■ 

. 13 


7. 


Number b£ areas 


/OO 


• 17 


.00 


8. 


Administration 




• 1/ 


n/. — 
-.0^ 


— . UJ 


■ 9. 


Applied research 


-.01 , 


.19 


-.05 




10. 


Basic reisearch 


— • PJ 


1 1 


• Uo 


— n9 

— . U Z r 


11. 


Clinical pr^^ 


— 1 
• -*-•/ 




. ±-i 




12. 


GuidahciB and counseling 




— ino 


n 


-.11 
• J. J. 


13. 


Teaching 




nn 


11 
• J. J. 


13 




Preferred Area ' 




' ' - 






14 . 


Administration 


.11 ■ 


\ -.06 


702' 


.08 


15 V 


Applied research. 


-.10 


.03 


.03 


. -.01 


16. 


Basic research 


-.10 


v25 


.08 


-•15 


17. 


Clinical practice 


-.OA 


— •Oo 


- . U / 


. U J 


" .18. 


Guidance and counseling 




— • ±7 


ns 


— • Z.D 


19. 


Teaching 


.19 


-.05 


-.07 


T35; 




Self -Appraisal 










20. 


Me an\ rating 


.17 


• .14 


T08 


.14 


21. 


lOciowleidjge; of psychology 


.16 : 


.28 


.06 


-.02 


22. 


, Knowledge of statisticis 


-.04 


-•10 


.07 


.03. 


23. 


ExperimentaX design 


.17 


.22 


.08 


.20 


.. 24. 


Research interpretation 


.12 


.09 


.07 


.07 


25. 


Clinical ability 


.10 • 


-•11 


.06 


. 06 


26. 


Teaching ability ' 


.02. 


.V-12" 


-.02 


-23 




Professional Activities 










34. 


• Number of activities 


.30 ■ 


• 19 


.24 


.tM 


35. 


Meetings, subscriptions 


.06 


, .16 


.23 


'.09 




Publications — 


. .23 


• 06 


.26 . : 


.17 




■ . 4' 


48 
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Table 14 (continued) 



Questionnaire,. Variable 


FH , 


EP . 


SMP 


MC 


37.. 


. ■■. ■ ' , ', . ■ ■ ,o " 

Planned , [did, indep , research 


.16 


.19 


-.06 


.08 - 


38. 


. Diid collaborative researcK 


ll2<? 


' .21 


.ii 


. .28* 


. 39 w 


Taught uridergraduates 


.14 


-.03 \ 


■ .14 


^-.03" 


40. 


Advised oh statistics 


: .03 


.00 


'.10 


.12 


• 41. 


Helped prepare book 


.14 


-.24 


.01 


.12 


42. 


Worked with equipment 


.34 


■ .17 - . . 


.13 


.15 




Other 










i. 


Has MA rather than BA 


— .26 


/;09 " 


.07 


.01 



\^ Correlations based on 3-item tests. Coefficients significant at 
the 5% level are underlined. 



m 
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Table 15 

Correlations of Number of Uausual-High QualiW Responses Scores 
with Questionnaire Variables*" 



Questionnaire Variable 


-MIT 

FH 


EP 


\ SMP ■■ 


MC 




btuaent/ UepC • C}ua±icy 






\ 


' r> , 


9 

4« 


irj.ans rtiu raLner xnan 


• lo 


.12 


.18 


# .17 


p • 




1 o 
• 1" 


on 


n X. 

- . 04 


.16 


L 
f • 


Pv" ra/lii o f- £1 PP A 

V7 IT aa u3 L c V /\ 


91 


. u/ 


. UD \ 


. U J 




AcccaQS accreaiccQ^aepu. 






^ ih'^ \ 

. U J \ 


no 








1 1 
. XX 


■ 1 9 \ 
. XZ« \ 


~- • yjij 




o3L ISI- Icu . Wl LU QcpaX^ Llucn L 






— n ^ 

— / U J 




J J. • 


uepar^iPBriL cjuaxicy inac^ 


« JL\J 


1 6 

' o xo 




on 




Qimnr^i^H ^Vvi^^imn 'n0^y/*nV^ 1 r^ci\y 

ouppoxL .Lnx^ougn psycno-i-ogy 


1 S 
• ±J 


9Q 

• zy 


. X J 


. ux 


33. 


Support through fellowship 




- "^.UX 


— . xu 




' 6, 


Program Eniphasis 










Plans acadeitiic Drocrain 


05 


14 


18 


19 


27. 


Prog ram objective : practice 


-'•15 


^ -.26 


-.21 


-.16 


'28. 


< Deeinnhaslzes teachlnc 


-.08 


-^.02 


• -.20 


.24 




■ • • Tt^#"ov"o^ ^ A T* o ^ o 










7 


. iiUIIlPcX . Ox , ctLcclO 


. u / ■ 


97 


— 1 

. X J 


in 
. xu 


8 


AHm "f n "f ^ t* t^i "f on \ 


03 




- 17 


• w. n6 


9. 


Applied research \ 


« 09 




- 06 


ni 


10 • 


Basic research A 


-•08 


.32 


.05 


03 


11. 


. Clihica]^;ij;>r^ctl \ 


-.14 


Til 


-.12 


-.03 


12 . 


Guidance '■^aiid counseling A 


-.08 


-;06 


-.05 


-.10 


13, 


Teaching 


.17 • ■ 


-.01 


r.02 


.23 




• V : Preferred Area 










14. 


Adminlstiration j 


-.19 


-.10 


. 02 


-.11 


1 s 




-. 01 


no 

.02, 


1 o 

-.12 ^ 


-.11 


16 


R^CS TP TP* All?f*h ' ' 


-.02 


.18 


.13 


.16 


17. 


Clinical practice 


.08 


.01 


-.07 


-.06 


18. 


Guidance and counseling 


.04 


-.16 


.03 


-'.20 


19. 


Teaching ' i 


-.02. 


-.11 


.07 


.26 , 




Self-Appraisal 










20. 


Mean rating 


« .23 


.12 


-.11 ■ 


• -.05 


21. 


Knowledge .of psychology 


.23 


.33 


* -.05 


-.11 


22. 


Knowledge of statistics 


.16 


-.08 


.00 


. -.07 


23. 


Experimental design 


.08 


.05 


-.10 


.14 


2^. 


Research interpretation 


.21 


.04 


.01 


.07 


25. 


Clinical ability ' : \ 


.02 


-.03 


-.25 


-.23 


26; 


Teaching ability 


-.01 


.08 


-.03 . 


.05 




Professional Activities 










34. 


Number of activities 


.20 . 


.li 


.18 


, .15 


35. 


Meetings , subscriptions 


.01 


.13 


.02 


-.04 . 


36. 


Publications 


.30 


.01 . 


.15 


.07 
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Table 15 (coatinuied) •> 



Questionnaire Variable 


FH 




SMP 


■ , Mc ':. 


37. 


^Planned , did indep^ research 


,12 


.04 


-.07 


-.02 


38. 


Did collaborati^ 


,12 


-.01 




: ■ , .24 


39. 




.02 


' .07 


;18 


: .06 


40. 


Advised on statistics 


-.04 


.01 


.00 


-.03 


41. 


• Helped prepare book 


.10 


■ -.15 


-.03 


-.02 


42. 


Worked with equipment ^ , ■ 


.24 


. .15 


.' ,17 


.22 




Other ' 










1. 


.,Has MA rather than EA 


-.23 


.04 


-.04 


-^11 



a 

Correlations bas^d on S-item tests* Coefficients significant at 
the 5% level are underlined. 
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\ ^ ■; Table 16^ 

Average Correlations and Significance Levels 
Combining Data over Four Tests-^ 



Questionnaire Variable 


Score • ' 


: Niuaber UnusuiSji Unusual -High 


.Student/Dept. Quality 

2. Plans PhD rather than MA 

3. Uridergraduate CPA 

5; Attends accredited dept. > . 

29. / Rates dept." high in quality 

30. /Satisfied "with department 

31 . Department quality index 

32. Support through psychology 

33. Support through fellowship 


.18** .18** .16* 

.20** . • ..17** 
.17** .15* .15* ' 


Program Emphasis 

27. Program objective; practice 

28. Deemphaslizes teaching 


.14* 
-.20** 


Interest Areas 

7. Number of areas 

8. Administration 

9. * Applied research , , ; 
* 10« Ba^lc research 

11. ; Clinical practlce'^^ 
. 12 " Guidance and counseling 
13. Teaching ^ 




Preferred Area 

14. Administration , 

15. Applied research 

16. Basic research 
17« Cliriical oractlce 

18. Guidance and counseling 
19 • Teaching 




Self-Appraisal 
. 20. Mean rating 

21. Knowledge of psychology 

22. Knowledge of statistics 

23. Experimental design 

24. Research Interpretation. 
^25. IClinical ability 

26. Teaching ability ■ 


.13* 

.15* .17** ■ 

. -.13*/ _ ■ 


Professional Activities 

34. Number of activities' 

35. Meetings, subscriptions 

36 . Pubiicat ions 


.18** .24*** .16* 
. .14* . 
.18**. .-13* 
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Table 16 (continued) 



' Questionnaire Variable 


Score 


Number 


^ Unusual 


UnusualrHigh 


37. Planned, did indep» research 
; 38. Did collaborative research 
39 ♦ Taught undergraduates. 

40. Advised on statistics 

41. Helped prepare book 

42. Worked with equipment 


.19** 
.13* 


.18** 
.'l9** 


.15* 

i 

.19*!*", 


Other 

1. Has MA rather than BA 




Two-tailed probability lievels. 








*£ < . 05 








**£ < .01 








***£ < .001 

■ • - . \ 
' ■ , ■ ' . \ 






" ■ 
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APPENDIX 



GRE .QUESTIONNAIRE 




Have you. attended a graduate or 'professional school this academic year? 

■■■■•id Yes;-' ', ' \ ' . . ;■ 

. /2 □ No - did not ap^^ /' 
3 Q No - applied but was liot accepted 

AONo- have been accepted and will begin attending in the^ future 
5 Q No; - Was accepted but did not attend { 
. 6D Np - attended previoualy bit not this ficademic year ' 

7 0N6 - other; ■ . ^ ^ 



What kind of program have you attended or have you applied for? 



■ ^ 1. D Psychology, ^ . 

i2 □ Education . 
* TQ Other Social Science 

4 Q Professional School: 
. :5'D'Did :nojt apply 

6 D other i , ■ ' ^ ' • ■ - ■ - 



4hat is the .highest degree you now hold? 

\ 1 □ Bachelor • s (5 .A- , B /S . ) 
; \ .2 □Mastqv's (M.A. , M.S., M.A\T.) ■ 
A 3 3othei 



What is the wighest degree, you plan to obtain? 

1 □ Bachelor/ s 
-2 D Master's 

3 D Phil). 

4 Cj Ed .Er: . 
- 5 D D. Psy. ■ 

6 □ Other:" - 



Approximately what overall grade average did ypti receive for your under- 
■graduate •Wbrk? .,, . ,, '^J.!'" 

' iP A---' ■ -• . ■ ■ ■ - , : "• ' ^' ■■ ■ ■ 
. . z Ci A' ' . ■ ' . ' . .. ■: • * 

• 3 0 3+ . . . • . . - . 

- A u -B • »■ r ■ ' ■ 

■■ sTj i- - / ■ ■;. , r 

■ .6:0 C+; r- ! . ■ , ■ 

./;;-7;D c' , : ' • , 

B D.C- or lower 
9 D No grades 



I ^-ai 



- .-.-HI 
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Approximateiy wljac overall grade average have you received for your graduate 
work to dat«? 

•iGa 

' 2 □ A- 
3 38+ 
' > Gb 
5 



■ ■ • ■ . 

C+ 

8 '~J C- or lower ' . ■ 

OX) No grades • 
•10 G Have noc accendcd ^rnduace school ' 

the rehaintntr questions are directed' tgl-those who have attended • - 
graduate programs in psychology during .j4he current academic year. 
. if you are not in this croup,' please ^;r|)p . hfere and return the 
IjuestxqnnAire in the enclosed envelope] thank you. 

^ * ■ . ■ . ' 

Which one of the following best describes your, intended area of specialization 
within psychology? 

' 1 G Clinical ' - " ' - ' 

2Q Cognitive 

^□Counseling ' - ' . o 

^ Q Developmental 
5 p Educational 

; 6 Li ExpcrxTOental, Coaparative* or Physiological <^ ^ 

7 i J Measurement • * 
8 G Organizational, PersormeX 

9 G Personality . 
. iO G School ' ' 
ill J Social • ^ 

12 G Other v ^.^ / o ^ ; ^ 

A^ssuming you could obtain exat:tiy the position you «would like, to vhich of 
the following activities would you devote si^gnif icant^ amounts of your time? 
(Mark as many as dpply*) . 

Adniinistration 

G 'b. Applied Resejjtrch ' . " , 

Q c- Basic Research 

G d« Clinical Practice 

•G e« Guidance and Counseling 

Q f\*r Teaching 

\ Q g. Other , in or related to psychology : 

G Other I not related to psychol6gy*: _ 



If forced to choose one, to. which of these activities would you pfefer to 
devote all' your time? , ' . 

1 Q Administration t ' . 

2 □ Applied Research - ^ 

3 Q Basic Research 

4 □ Clinical Practice 

5^Q Guidance and Counseling 

6 □ Teaching ' ^ ^ 

This question is concerned with skills arid competencies within the field ' 
of psychology. We would like your assessment of your present level of 
achievement, relative to other students with a similar amount of graduate 
training in psychology. In each box place a number from 1 to 5, using the 
scale showt; below. ~ \ 

f • ■ ^ 

1. Exceptionally well prepared 

2. Above average ^ ^ ^ " 

3. , Average ^ 
^ Below average 

5. No experience or does not apply 

a. GencraL knowledge of psychological literature 

.Knowledge of the literature in your area of specialization 
Familiarity with research techniques in your area 
Knowledge of psychometric and/or statistical theory 
Ability to apply statistics to research data 
V. Skill in conducting ^perimQnts ^ involving human subjects 
. Ability to gain insight into the problems of clients or patients 
Ability to design original research studies 

□ i. Ability to evaluate research designs 

□ j» Ability* to teach complex ideas to undergraduates 

□ k. Ability to interpret research findings 

^ ■ 

Which of the following most characterizes the main- objective of the program ^ 
in which you are enrolled? - 

1 D To prepare students to be competent researchers. 

2 G To prepare students to be effective teachers 

3 U To prepare students' to be effective pi?cictit ioners in professional* 

■■ { applications of psycholn<»ical knowledge ' - * ,^ 

Which of the following least characterizes the major objective of the progtam 
in i/hidh you are enrolled? V 

1 Cj To prepare students to be competent researchers 

2 C To prepare stodents to be effective teachei;s 

3 Q To prepare students to be effective practitioners in professional 

applications of psychological knowledge 

: \ .. • . ■ H . • 

■ - ■ ■ / ■ . . ■ .: . • ■ . • ■ • • . •, • . . ■ ■ ■ '} 

.i^- ■ ■•■ , ■ • ■ ■ , ■ ■ ' . • . -■ • • .. • r 
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Overall, how do you think your department is ranked on a national scale 
araong graduate psychology departments? ^ 

,1 □ Top 5% - . ' 

2 □ Top 10% - . 

sQTop 25% . - 

4*D Top 50% 

5 □ Lower 50% ^ ' ■ „ 

Is the uni"ersiuy' you -are now attending; 
1 G Year top preference 

2[lX)rxe you are satisfied with, but not your top preference 

3 G One you are not really satisfied with 

If^you like'^ give th^e name of your university here: 



What has been your major source of financial support this year? 'I 
1 □ Fellowship . /\ . i 

.2 0 Assistantship or other employment in- research 

' 3 D Assistantsh.ip or other employment in teaching / 

4 Ll Other employmerft related to psychology 

5 □ Other > . • V ~ : ^ 

Which of the following have you done within the currentV academic year? 
(Mark as many as apply.) 

Da. Attended one or more meetings of a scholarly or professional society 

D b. Subscfil^ed to two or more scholarly or professional journals 

D c. Been author or coauthor of a scientific paper accepted for^ presen- 
tation at a meeting of a scholarly qr professional society . 

* \ D d. Been author or'coauthor of a scientif icx^aper submitted 'for 

publication to a scholarly or professional Journal' 

m ■ ' \ ' ■ ■ ' 

U e. Be'fe'n author or coauthor of a scientific pap^r accepte_d for 

publication by a scholarly or professional\^ounial 

D f . ..Prepared a detailed proposal or plan for a dissertation, 
master's, thesis, or other major research project 

lJ g. \ Carried out an independent research project \^ 

D h. Carried out a research project in collaboration witl^ another 
student or a faculty member \^ 

0 i. Had teaching jresp^x^ for a section of an und'erg^duate course 

G j* Conducted a section of an undergraduate class on one or several 
, occasions .^ \ 

■ ■ • . • . ■ . ■ ■ A 

V. . (continued) 
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17. (continued) 

Dk. Frequently advised or tutoried other graduate students on 
psychometric or statistical problems 

Dl. Assisted in editing of text or preparation of bibliographic 
material for a book 

D^v^- Desi^^ne^d. and built a piece of laboratory equipment 

Dn. Learned to operate or inaijitain a piece of electronic equipment 

Do. Trogrammed a computer to analyze reseaFcK~dat-a 

lo. We would appreciate any additional comments or explanations you have to make. 



Thank you for your cooperation. 



.... •■^ ;:■»-«« '^ligJj 

wm 
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